Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Batch asset invalidations #11937

Merged
merged 107 commits into from
Jan 8, 2025
Merged

Batch asset invalidations #11937

merged 107 commits into from
Jan 8, 2025

Conversation

somebody1234
Copy link
Contributor

@somebody1234 somebody1234 commented Dec 24, 2024

Pull Request Description

  • Close https://github.com/enso-org/cloud-v2/issues/1627
    • Batch asset invalidations for:
      • Bulk deletes
      • Bulk undo deletes (restores)
      • Bulk copy/move
      • Bulk download
      • This avoids flickering when the directory list is invalidated multiple times (once for the mutation corresponding to each asset)

Codebase changes:

  • Remove all AssetEvents and AssetListEvents. Remaining events have been moved to TanStack Query mutations in this PR, as it is neccessary for batch invalidation functionality.
  • Remove key and directoryKey from AssetTreeNode, and keys in general in favor of ids. Not strictly necessary, but it was causing logic issues and is (IMO) a nice simplification to be able to do.

Important Notes

None

Testing instructions

  • Ideally test all asset list interactions, especially ones that fire mutations:
    • Copy, delete, undelete
    • Create assets (optional)
    • Add labels (via drag)
    • Download

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

  • The documentation has been updated, if necessary.
  • ~~Screenshots/screencasts have been attached, if there are any visual changes. For interactive or animated visual changes, a screencast is preferred.~
  • All code follows the
    Scala,
    Java,
    TypeScript,
    and
    Rust
    style guides. In case you are using a language not listed above, follow the Rust style guide.
  • Unit tests have been written where possible.
  • If meaningful changes were made to logic or tests affecting Enso Cloud integration in the libraries,
    or the Snowflake database integration, a run of the Extra Tests has been scheduled.
    • If applicable, it is suggested to paste a link to a successful run of the Extra Tests.

@@ -224,7 +219,7 @@ export default function DriveBar(props: DriveBarProps) {
<ConfirmDeleteModal
actionText={getText('allTrashedItemsForever')}
doDelete={() => {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't it be a promise here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point. i was being silly and figured it shouldn't be a promise because the component doesn't take a promise...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found doDelete is passed directly to the Form component, so we can use promises, as the form will do all the magic ✨

Copy link
Contributor

@MrFlashAccount MrFlashAccount left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ✅

A few nits left below

@@ -503,6 +487,10 @@ export default class ProjectManager {

/** Delete a project. */
async deleteProject(params: Omit<DeleteProjectParams, 'projectsDirectory'>): Promise<void> {
const cached = this.internalProjects.get(params.projectId)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be avoided in the future - cache management is purely the tanstack query responsibility.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

honestly yeah. but i think this is more or less a workaround for issues in the engine API, the alternative is we'd have to implement localbackend through tanstack query instead somehow.

specifically: the "cache" here is so that we actually remember the state of the projects because (iirc?) there is no way to actually get the state of the projects though the Project Manager... if we could then none of this would be necessary

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But we have the same cache in the tanstack query, haven't we?

}
case backend.AssetType.file: {
const details = await this.getFileDetails(asset.id, title, true)
invariant(details.url != null, 'The download URL of the file must be present.')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think here would be better to throw errors/reject promise instead.
See: "You're fine throwing a descriptive error here because it's just very unlikely this will ever happen and even if it does you wouldn't really know what to do about it anyway."

https://www.npmjs.com/package/@epic-web/invariant#:~:text=You%27re%20fine%20throwing%20a%20descriptive%20error%20here%20because%20it%27s%20just%20very%20unlikely%20this%20will%20ever%20happen%20and%20even%20if%20it%20does%20you%20wouldn%27t%20really%20know%20what%20to%20do%20about%20it%20anyway.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason we don't just switch out the invariant package to get this for free?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYM?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we switch to @epic-web/invariant, we get:

You're fine throwing a descriptive error here

for free, right? the issue with the current tiny-invariant package is that it strips out error messages

both packages are just small wrappers around throwing errors anyway, right?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, possibly, yeah. Tiny invariant strips messages only for kb savings:

CleanShot 2025-01-07 at 17 04 59@2x

I think for us it's not a big deal...

@MrFlashAccount
Copy link
Contributor

🚫 QA:

  1. Moving assets in and out of the assets tree breaks the dashboard:
CleanShot.2025-01-07.at.16.42.59.mp4
  1. Perf regression: Refetching re-renders the whole Table
CleanShot.2025-01-07.at.16.50.30.mp4
  1. I think for the assets table it's better to invalidate both active and inactive queries.

@somebody1234
Copy link
Contributor Author

I think for the assets table it's better to invalidate both active and inactive queries.

just for delete/restore? or for move and copy as well?

@somebody1234
Copy link
Contributor Author

Refetching re-renders the whole Table

whoops... how were we avoiding re-renders previously?

@MrFlashAccount
Copy link
Contributor

whoops... how were we avoiding re-renders previously?

AFAIR we didn't pass the visibility object to every asset. You can use Profiler in the devtools to see what props/hooks trigger changes

@MrFlashAccount
Copy link
Contributor

just for delete/restore? or for move and copy as well?

I think yes

@somebody1234
Copy link
Contributor Author

I think yes

yes as in all of the above?

@MrFlashAccount
Copy link
Contributor

yes as in all of the above?

I think that any mutation that affects assets state should invalidate all 'listDirectory` queries (both active and inactive)

@MrFlashAccount
Copy link
Contributor

@somebody1234 Re-renders issue seems like not fixed :(

@MrFlashAccount
Copy link
Contributor

CleanShot 2025-01-07 at 18 59 59@2x

Seems like the regression caused by un-memoized columns list

@somebody1234
Copy link
Contributor Author

Re-renders issue seems like not fixed :(

yeah my bad, i had fixed it before and verified it, but then it re-regressed because i was too trigger-happy with removing useMemo DX

so it turns out react compiler doesn't quite work right sometimes. i tried removing useCallback in the useEventCallback which works in the playground but not in the actual codebase i think?

@MrFlashAccount
Copy link
Contributor

MrFlashAccount commented Jan 7, 2025

@somebody1234 it did memoize hiddenColumns, but didn't memoize columns 💀

CleanShot 2025-01-07 at 20 12 47@2x

There's clearly something with the usage of the columns variable: https://playground.react.dev/#N4Igzg9grgTgxgUxALhAegFQYAQAEBmAlgDYLYAqAhgEanYAmhYADsZQJ6EB2A5tpdmJMALtgj5szGBABWCOMLAA6bBjQAdLoQC2zCDFHBN2bAGEAFiXowEXADTHscYhC4IAoqW23hDriaYANUohek8Eby5fR29tCD8TMGFKA3IYSi4wQmFCVwTsKDAPfHx5aP8CooBJXQR0nIA3BAAJDPpSfMKEACUEfE6igGVk4QR84XZmMgARJmZKYThzccmZ9J53JqiVqewAaQR2aggUsK3ykwndgFkoZNoevoB5ajkFHbJr6CLNn34wbC9SgKL5dX7bRxXMhAhQAOQg9DGkNWgL6H1R+Bebwu2Ch2EGCGEwwWCAAggpcvZNABfbD4aTabAAchswOETM0mh0egM2GAlQQtxGlLsAoAilA6uxTEI-rT6RBGUzcMlMsk4ABrNCshQAWgAjpKYOwOVpdPpDLiTklsPKGcydcJdcJrTl8CauebeThKACAF50+1Mv0I02enmiabSZgAMRICDANUoPDICqVAGI0L6ioo0NYIMwAPpEUjKMANHim7kWvnYOOkNKEHgpmCi7wwFMABWjYFFXW7CYBdsVzMzcEVejcUTAWZghEoVa9loAQncXfYKAgAB6iYcZtDj81T3OkueUUwT1w+MCLiO4lH80lgHPdCAAdyqXDcMG7BaHgZHJkx0vY8Z3oX1zGOU40CfF931vGtH2fQlXzfW0AP3Q9J2vPMIKgmB6Bg5DhFQhDeX5KpYSqcgqlJAAZQtuieAB1QtBnIUlyHcdC01HA8QJw8CwEgk4CKIuC3yzYjUIAVRyYgb3DGs8X5QYLRoUgL2IKBtH8Pc+Kwq9p1w4T8MI8dtN0-jLK4OSSEUs07yMCotJ0qkXKeOiZOuWFC1MQZBj8ujSQC-JTE87zfKqcLYUGMKIp8tjmhYwsuIADXIQsqmmfJpncGNSRkujMvcWFSWXOj3GmPyEti-IU2EVzdLokQ-H0oD+KPQS8NE8yIBs6y3LshSyNEWFKG8Jr-F4jrDNAkyROgiy3LQcbJv6tzRtrcKvMS5p3FJaZKIAcR4oNgK64yhMWsTlt05oEEoRheC2uiaAQYgMIMgSrp66C3uoD6tv5dwYGkGBZhYNh2DOwCLuw4zQfB5doC4cDjWB7AqkgNhRje9hoF3L7Zp+3Nsf6kl8cJzGCVIClXGXGBCmWe9dieLgo2TTsUgm-8Zvhozc1pspKUZ5nMc-REiC0UZBmYbhv1hzDSZnOWFbqLaYwgOBClPR7iYF+aknYUhCK1nWwD1hclN5QYK2uX0NQNzqEaF+3HcxkLBncchAuuKprncVKAE1O249rM3A5I0G0HQEHIVYHOrciBVJO4IEGOBpGIT6I7QcwIAgDUZ0odPM+z4hmkL4vgccJ76FgwkwABj6wCFBZKSeZgclcXtHHHZh2EbxR257rgu7HvuKjiJph7bu4O9cCfKSnkwbBnsliOb96FNHzvu5XtrnYLouZ2oYENVsehlwWJYEHoKvT8x8-NSvvel4P3u+yKG-X7RiUpRKz4ifYuaAX6XzRo-GuNtLRdBkqwE49B6wJiAR1EBZ8L5X3gS4J6yCwBQKTkuWsXRTB3FJGjbmSQyB53QQeMhFDfSjAIZjME5xTAhGIOA1BmZaEIDYRw8BzCYHEOqFEOoRR6ZcG6Ivbh+dq4zm4KMGAEix5CMcohAUXdbC-mxLI2hUhZBlDUcnWBQwXQ2D0fItASR9AtHkSwoY7AuBwF6BIGhViwBOJcX0YxRD+RdHIK6ch9A6IQD4O40+aAXSMOCaEngvinJ12ItzNwxBhi2IGI8HMw8UkfV-MwVeAoCTCByRkPJPZMnFOxqU1J5AIg8hSCQdggQmCEAeEffmaBoaExnDUoGwi8Q+gBNmQkBIUhLBvjAZ23S7i9OImM+A5hJkOIQMUwYUBmwJknrImZJ55mPUWcs4Rc8qAPAvGInc1xbBQGmRwHp4km6nM0q4UYlzrm1wqHADIaQMhgFKDAZchI3wIFsOw0YPB9CEATPkJgoSvnEDBQgCFxp0SIuRewDp50ul3NmWgNF+h2CDDfNkO+MA8UknRSsk5GkEBVFGNoPmWLdkzijIQJoUkcxgCebS+l+D7HCP8UUYeaQQWijxLMGwCgCUAONDsnFuZWXspGcIEVdin4CoFBKsoBKqj0EZXDbFBNcWKoQHmQgkrzHsF1Xy9V6jeScx4F8cCn1OlxGdSy9YTqQhbXgUUAwBIs6Ei9S6rFbqQgzl9XUIk8gbDCGDSsmMUAc4yT9QSZ8lJZEGIaIQREyiYJ3HML+bNuaPkmC6L-CB9BMmzBCYg7gPAdFlAdlwZMdQYyUBIJk3o44vxlECY2kEGRW2thpM7LNObxFoArVfItE6YCls1XOJo6SbCZPhIiB2zBKmEnYVwUwrJRhz23Y1DI0x3xcBwVWxwXRiktwUg67mJtEHHthAgYFSQtbEFzbql9CJBSUC3deoYhJKGjGmAsSgx7hYKHvkeoDqzCRUA7ISLVUqUXweKS0sAUAOGEo+mUe+Bx2CFICWEngpBUOWvcFueYmQRTIl2NB0YDdiKfnwPEUdnTx25o9WyhAs6S0aq6J+ZgdxlzcGejwfV+5uOTpE2JiT9awACY1kJoocKQgrtbZm6QxbJ0abSeY1tKn51qYQ3GhEIQdMQD03m4NJmVnjWzTwBY+gABM0xrO2ZnE5psrmYAeYc2Zt6UBnHmHvgOxQXm50zki8p3Tc6Vl1J3NFnjaBkvCCC3a0QeJp1o2dn67NiAZx5foFtFS2Bh66tkYVwgxWp2YLRgu5V2NIv5GFasfIpWE5THqoSHJdRY7ptcGtJEFQqiBSeGHKisJjqFieN0eb02qr5E7GwdylwUTkKHsRdElGCXDz2+a7Vxof2caxbV+rpXysPkq8RGV7AiOyLuCQbIUK5k5ge1tYeD3nYvaEDkBMDzhBfYGVt4iqr12pixf9t7QPhU2AQFDm7uxHxcB2zmSHf7nvyThx9xDiPkdg9R3diSYAH0cEvTj17gOwLrE9gFH2jEWJxWwMFZc7g6Ks5zA6pMKZAnLnW07POsPad5npxqksHgdy2CyK4anAP3toCl2xiAmMuCF2YArvHyvQuSMIU5bAeUCpFUys0UksJpiVW6NrsX3BRPCHE2jJTr1taaaMymP7uOxcGa0ymBdXyuCdkG0wOXXAnWEHdPt405zRjTlWyH4bXByRj3GMaOMaNab4GDzAIbYfMUGtF0rqYufQ8r0xvQbWOk5Re5p0rsAGQ3t+nvuCFVKQGoG40cS4QhabBFGcdQ2viugc5hR2QVSFpVe2-rxaetmNhBvgAFIQG4GKt8VzkPT6B8kEgxK0Yb-98IrDbSacw06UXoHx-qCn62twBojSMiiBmjkdHuo78P6iGGNG8g2AWLdUmsgDqC-GcX3D3BAJkPkRwTAHANQRwRROofAYEMgUA-QVtcDZISAioNeR6SvC9GGWwGlegKaMAZAbAVkXA4gGGKaAAbQAF1HBqQaROQuAUD1gEAlAbAeARA6giMAAKJkAgh4IgjaXSG8UUZyRIO+bQSgUgv0JQFtRoDwLgHSXgqaAASiUBSHSHYF4I0PINcEoN0LajUOYOgOMBwCqAkB7zIHqFyDEAkHgOURFnl0BgXxBX8GsOwGkO4AYGjBDDcFxHMAWGwGEmgC-WwEBmwGzSyHaX4Hy2sPMJCKzn6jYAeCcBeQ7W-FFCYAiI+nfECJyPv20jGECNsFKIiIvigC13zGYH8LIGJRzlyJCILjfC4CUHMI0GcV7lEH9iomuG8kLGmCYk7AAC0nhYQg5KIuJuhvZTAaJxjGJOIqgnhsAABebAAABiUAAFZNAzD-AcByBwtsBwsmxzBRBxBsBHolgyD8ifDPD7g6Bjh6B2AVB+j2ImiJ0og6srMXRyiijJR-gQjCQIiYYEiDiKAO1iA996AQiJghBeB2iDjOie0bQmJWJ9oqhjpmhMpOw0o1jsAABmAANj2KwAoGOKyGb3sPKMvXrRCPll7RgBUFgK6LVDZyeEOhOjYk7EogmMW0GCqBGKDjxIJJJOYNRNEEGBkmOmOncHYmWNikLBjAW0LFhCeFIOVQWQmRSCUHWU2SSEpDoIJOoMcAkJMGwEvnYFIKZA12QFSKBnyGwO-xgFIN0LWIAD5mQ7SHTiAmQnT+B6B6BAkHs3TDQpQ1DPTsBwzjRNCgzAk3okg6lc9eD+QNcSDsBqCmRfSmRaDbQ1CAzERSBRgYwGRQzsBeCYz2BIzVivSqylAizCQEBSzFREyVVBtUzsB0zSCsycy8zqQCyGD8hzSTArSbS7TEQwAs5CBP4uB-THBnTc03SayvTbSIBkBJzpzZz5ysDAzgyIByzKyjRqyoz6z64EzGFkztBOzuzMymRNy5xtz+zBzdzGySyyzjywzjyVzozjyGz8N3zWzLyOy0yIAMysyHyZyx5cz8z8hqQ-B6C2SbRpTZT5T5ilSVTFtzdBhNT9lxklldT9SUxDTXBjT1jTSKgRzLTDgbSgiSCcyAybAXTlyoymQ6L7Sd4dyLS9yQzPyKyqyfyzz4yIA2yrybykUO4mh4RwLszOLnzCyALmyPypQvyIzTy-y3ylKgKkyQKuyJLFDpKezZLAY-T5KhyzSFzqLrTmR2LILtzGKr46gWLaybLfQNyEwtzoKAzzyDy+Kjy1KXKhL9zRLdK3AXMDKwKjK7LoKzLXzFKWztBDyBL1KpR-ziytLtAQqUy0z9K+NDK7zorKQYKBy4KELNBJT8QZS5SFTxjApMKQ4w5cKcxtSCLmSiKtkjS8zyKLLdyxzmQoRkADFsQuKLSmKlyKyfymQBqhqygRqTAfKkrvyUrYyfKsrrz+QoQZKZqFBiqXzuLNKErFqAq6yNL4qGQ1rOzNqjLtr2RYqTB4KeruK+qprVhkB2Mv0NYHLmKJrWKBr3qS1vLhKjrjRBK-zVrgLsrWYEwjL-qNY7qLSDrlLjRVKQblrXjEbtL2zIarq7zYb514aHrKLLLnq-r4w5qyDHLXSfqXKXqpg3qybAb9zgaTzAqwbhKLqNrE4Yayb4aTAMbEq-LkrWbUr+aOaoaZKpddrSrHqLSSbXqJFY1yaxqnLqbVyBqFbCRyaFrBalrhaVr2aIb1rxajKNbbrYLLL+bmbQaRazrMaxLOapgZLTapbzKibeqaL+rXqo4QhuANQlbKbnK1avaIMES-bGbeKVL+LdaTrUrwadLsauaCqQ7faXa4r0rDqdbjrfybb07zrDbLrE6ILk6uA-aCayqkKpSqq0LFS6rVSJjjoljAgg5yBQ53AmrRkDkdS2qNliKx4yLMyZbRyPbaaEBBrpBhqvrxr3Sabprx7Zrw7fLI7-LUa9bXi46sajawrJL45C6mQbrU79rbaBal6haY7YzRb86crwq+Metoa7z96y7B6rKbS-r+oAbLLlaqbp6g66a8atagbM6V6z616Db47N7cqmhb6Ja364bza06myM6T7o7s7z6j6xat7FCoGYaYH8a4H7rhzibh7SbSB-bvrv7Pbf6GbLLtakGs6gqLywHxLr7IHd7JbeaGAj6ra0a0qEG87GGr7t6sHcaea8HbQCH3brKR6Nzi6w6P6A7VaKHR7vbQ7-6mbAGWbgG4zgrL69LmGd7HaoqZGD6EbOH1HrbUHc67bQqIH9G76i7kgVHH6uBEKySLCuB2Nc9F5-AbBcZ74rR+BfDkxgS6Yx4WTOjHDEDEAjd1gmNKQp9zT9C8DsBuA3sQhJZtxSDlDtBAYYBHBEnKC4SUhhBMmdIcm8mcCDD8C0YSnsm6gmCuBXH8QRgyB5hkJYSe9pANlzAvoBBgBcBQ7SdHkaVaQ-i+FAFEAc4wnNBtw7wImkDBnFBuViRRhMDsCnpKmKi-56BSDStyn1mknpAIBhBo8rVtmjdjs0NTm9mKCYYpyK5Y9Mi6hXFSDXEsQygAAeHE64SqLwHwD065jZr5cFAlUg-FY0AFpJyAAwVXUgtSaF9xiAd5uFx4hAKaL0gAHy7KTWIAhYKeasnwRbdKheEBhfxAJfYyRfUjOREK4AxaxZzh-IaBX3oFxZhirNIJ+2PNZeBJBz4shnmEWHMCRdGWaZT0pHec5alA9P+YqHyZhg1w3QAxeYqbwPefbgeFedeA+aBBuc3QldYyrUq3RwRxBSh2lZlbWZuZOInSmjdLui4FBZpcZeZe5crwvEHkDqiJddlZVYKbdbuE9aZZzVdYgFA1HorLcDfG5iYuECI1IJOZ-T0qjZSB8F1XjYuctV1WdeDZ9f2YKYamHih2XFObdJzQ5YNZ-O2xNaR2x0xeUJznqcafQIEGzG1nnGY2wG726d6f6d9oWdQngoFFhI8YppdPrSmbcBoxrDmaieHlQmWfH25aYHcEYBfx4FG1IOOH6kegaa4EYN3egOwHyQBBHZ7YGepQeGpAnZmenbERgEibIAvdIGPdWYprzZhksCDNsE3cLlIAyG5fZYWYe25c+z5bmFviFbWVFckX1dA6lYtbfataBaRRBbMApQJSXZlnnGIEiw3bhLnF4GwDrexe5eVWbVbUiBKXlmeYxFg6DSHRTEo9JHlhlf3caaOMfd5LpH0C8IY-pOVQBHuOOJRevand5BncfeInI8Y58GY8IFfblewALeIlLbOaq3oEreNeIiI-pZxdzateyRU4rLLYWcTeVQ5a05zGzZZb3dMPJI49xBpRpJuv+E5QnfwD1zHgWa5RpV4IMQKXLc5W5WPcjPNIqv5E-cRA3CrNFDg5RScHQ-i7I4Y4iFk+o76HQnWP87AH7m6NrBSZyBCFw4mkHyy57GYNy-ZPtYBHWIaimhaiSF4K6FbE2crSUChFFGQ-RQ0JLCUV4N4PtZXMssENNmIKUDooG6dfyBMN3ZMAqsi9BRpZq6U53RpYa+ECa79VFHASvna86wS+BZBqUF67qF4Mssm5sh-IAEIRv74xuJvBvpvmDsDhBYB-B3nGAGgPTgBqvaRgAFu90lvqR3mzUvvG2uAQBqQgA

@MrFlashAccount
Copy link
Contributor

QA ✅

@somebody1234 somebody1234 added the CI: Ready to merge This PR is eligible for automatic merge label Jan 8, 2025
@mergify mergify bot merged commit 25933af into develop Jan 8, 2025
39 of 41 checks passed
@mergify mergify bot deleted the wip/sb/batch-invalidations branch January 8, 2025 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI: No changelog needed Do not require a changelog entry for this PR. CI: Ready to merge This PR is eligible for automatic merge g-dashboard x-refactor Changes that should not be visible to the end-user
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants